Method and electronic device for generating partial virtual model of objects

ABSTRACT

A method for generating a virtual model of objects is provided. The method includes detecting, by a first electronic device, a communication session with a second electronic device, obtaining a first set of objects displayed on the first electronic device and a second set of objects displayed on the second electronic device based on the detection of the communication session, determining a first object from the first set of objects to be mapped to a second object from the second set of objects, predicting attributes of visible portions of the first object and the second object by mapping the first object to the second object, obtaining depth information related to the first object and the second object, and generating a virtual model of the first object and the second object based on the attributes of the visible portions of the first object and the second object and the depth information related to the first object and the second object.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation application, claiming priority under§ 365(c), of an International application No. PCT/KR2021/008818, filedon Jul. 9, 2021, which is based on and claims the benefit of an Indianpatent application number 202041031596, filed on Jul. 23, 2020, in theIndian Intellectual Property Office, and of a Korean patent applicationnumber 10-2021-0080202, filed on Jun. 21, 2021, in the KoreanIntellectual Property Office, the disclosure of each of which isincorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to a virtual model generation system. Moreparticularly, the disclosure relates to a method and electronic devicefor generating a partial virtual model of objects.

2. Description of Related Art

A user of an electronic device needs realistic viewing and interactionwith remote objects or another user's to determine feasibility andcomfort in using the product features. In existing methods, the methodincludes receiving audio and video frames of multiple locations havingthe another user/remote object at each location. Further, the methodincludes processing the video frames received from all the location toextract the another user/remote object by removing a background from thevideo frames of the location using multiple cameras. Further, the methodincludes merging the processed video frames with a predefined videoframe to generate a merged video, so that the merged video gives animpression of co-presence of the another user/remote object from alllocations. In another existing methods, the method includes receivingaudio and video frames of multiple locations using multiple cameras.This results in consuming a large amount of resources (e.g., processingpower, memory, battery, central processing unit (CPU) cycles, or thelike) for processing the audio and video frames.

Further, existing method does not use any dynamic intelligence topredict dimension of parts of the another user/remote object which isnot visible in the multiple cameras. Further, in another existingmethod, a time of flight (TOF) sensor generates a complete 3D model evenneed of complete model is not required. There is no mechanism ofgenerating only partial model of objects as required by a user.

FIG. 1 is an example scenario in which a user of a first electronicdevice is speaking with a user of a second electronic device about aproblem of using their wheelchair bought over a video call, according tothe related art.

Referring to FIG. 1 , consider the scenario, a user (102) of a firstelectronic device (100 a) is speaking with a user (104) of the secondelectronic device (100 b) about their health condition and problem ofusing wheelchairs (106 and 108) as follows. The first electronic device(100 a) “Hey dude! How is your recovery going on?” Second electronicdevice (100 b) “Hello Tom! The recovery is ok, but I am using thewheelchair (108), the wheelchair (108) is so uncomfortable to me, and Icould not even fit in properly”.

First electronic device (100 a) “The same thing happened with me. Eventhough I have selected this wheelchair (106) over a video call. But thisis so uncomfortable! I wish the video call could have helped with thedimension to buy the wheelchair (106)”.

Referring to FIG. 1 , the existing method does not use any dynamicintelligence to predict dimension of parts of the wheelchair (106 and108) which is not visible in the multiple cameras. This results ininconvenience to the user.

The above information is presented as background information only toassist with an understanding of the disclosure. No determination hasbeen made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the disclosure.

SUMMARY

Aspects of the disclosure are to address at least the above-mentionedproblems and/or disadvantages and to provide at least the advantagesdescribed below. Accordingly, an aspect of the disclosure is to providea method and electronic device for generating a partial virtual model ofobjects by capturing only visible portions of the objects in a videosession. This resulting a low resource usage (e.g., CPU power cycles,battery, memory or the like) of the electronic device for creating thepartial virtual model of objects.

Another aspect of the disclosure is to generate the partial virtualmodel of objects without requiring any special efforts required from auser to predict dimensional mapping of the objects while creating thepartial virtual model of the objects.

Another aspect of the disclosure is to generate the partial virtualmodel of objects by capturing a visible portion of the first object andthe visible portion of the second object and predict hidden dimensionsof the first object and hidden dimensions of the second object. Thisresulting the low resource usage of the electronic device for creatingthe partial virtual model of objects.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

In accordance with an aspect of the disclosure, a method for generatinga partial virtual model of objects is provided. The method includesdetermining, by an electronic device, at least one first object from afirst set of objects to be mapped to at least one second object from asecond set of objects based on environmental observations. Further, themethod includes predicting, by the electronic device, a plurality ofattributes of at least one visible portion of the at least one firstobject and the at least one second object by mapping the at least onefirst object with the at least one second object. Further, the methodincludes providing, by the electronic device, the plurality of predicteddimensions of the at least one visible portion the at least one firstobject and the at least one second object as input to a sensor togenerate the partial virtual model of the one first object and the atleast one second object.

In accordance with another aspect of the disclosure, an electronicdevice for generating a partial virtual model of objects is provided.The electronic device includes a processor coupled with a memory and apartial virtual model controller. The partial virtual model controlleris configured to determine at least one first object from a first set ofobjects to be mapped to at least one second object from a second set ofobjects based on environmental observations. Further, the partialvirtual model controller is configured to predict a plurality ofattributes of at least one visible portion of the at least one firstobject and the at least one second object by mapping the at least onefirst object with the at least one second object. Further, the partialvirtual model controller is configured to provide the plurality ofpredicted dimensions of the at least one visible portion the at leastone first object and the at least one second object as input to TOFsensor to generate the partial virtual model of the one first object andthe at least one second object.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 illustrates an example scenario in which a user of a firstelectronic device is speaking with a user of a second electronic deviceabout a problem of using a wheelchair bought over a video call,according to the related art;

FIGS. 2A, 2B, 2C, 2D, 2E, and 2F illustrate example scenario in which afirst electronic device generates a partial virtual model of objectsduring a video call, according to various embodiments of the disclosure;

FIG. 3 is an example scenario in which identifying points of structuralintersection is depicted, according to an embodiment of the disclosure;

FIG. 4 is an example scenario in which determining attributes of notvisible part of an object is depicted for generating a partial virtualmodel of the object, according to an embodiment of the disclosure;

FIG. 5A illustrates various hardware components of a first electronicdevice or a second electronic device, according to an embodiment of thedisclosure;

FIG. 5B illustrates various hardware components of a virtual modelgeneration controller included in a first electronic device or a secondelectronic device, according to an embodiment of the disclosure;

FIGS. 6A and 6B are example sequence flow diagrams illustrating step bystep process for generating a partial virtual model of the objects,according to various embodiments of the disclosure;

FIGS. 7A and 7B are an example flow chart illustrating a method forgenerating the partial virtual model of objects, according to anembodiment of the disclosure;

FIG. 7C is a flow diagram illustrating various operations for generatinga sub-partial virtual model of a first object based on a plurality ofpredicted attributes of a portion of a first object and environmentalobservations tracked in the video session, according to an embodiment ofthe disclosure;

FIG. 7D is a flow diagram illustrating various operations for generatinga sub-partial virtual model of a second object based on a plurality ofpredicted attributes of a portion of the second object and theenvironmental observations tracked in the video session, according to anembodiment of the disclosure;

FIG. 7E is a flow diagram illustrating various operations for predictinga plurality of attributes of a visible portion of a first object and asecond object by mapping the first object with the second object,according to an embodiment of the disclosure;

FIG. 7F is a flow diagram illustrating various operations fordetermining a dimension of not visible part of a first object, accordingto an embodiment of the disclosure;

FIG. 7G is a flow diagram illustrating various operations fordetermining a dimension of not visible part of a second object,according to an embodiment of the disclosure;

FIGS. 8A, 8B, 8C, 8D, and 8E are example scenario in which a virtualassisted shopping using TOF sensors is depicted, according to variousembodiments of the disclosure; and

FIGS. 9A and 9B are example scenario in which an assisted fittingmeasurement using TOF sensors is depicted, according to variousembodiments of the disclosure.

Throughout the drawings, it should be noted that like reference numbersare used to depict the same or similar elements, features, andstructures.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thedisclosure. In addition, descriptions of well-known functions andconfigurations may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of thedisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of thedisclosure is provided for illustration purpose only and not for thepurpose of limiting the disclosure as defined by the appended claims andtheir equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

As is traditional in the field, embodiments may be described andillustrated in terms of blocks which carry out a described function orfunctions. These blocks, which may be referred to herein as units ormodules or the like, are physically implemented by analog or digitalcircuits such as logic gates, integrated circuits, microprocessors,microcontrollers, memory circuits, passive electronic components, activeelectronic components, optical components, hardwired circuits, or thelike, and may optionally be driven by firmware and software. Thecircuits may, for example, be embodied in one or more semiconductorchips, or on substrate supports such as printed circuit boards and thelike. The circuits constituting a block may be implemented by dedicatedhardware, or by a processor (e.g., one or more programmedmicroprocessors and associated circuitry), or by a combination ofdedicated hardware to perform some functions of the block and aprocessor to perform other functions of the block. Each block of theembodiments may be physically separated into two or more interacting anddiscrete blocks without departing from the scope of the disclosure.Likewise, the blocks of the embodiments may be physically combined intomore complex blocks without departing from the scope of the disclosure.

The accompanying drawings are used to help easily understand varioustechnical features and it should be understood that the embodimentspresented herein are not limited by the accompanying drawings. As such,the disclosure should be construed to extend to any alterations,equivalents and substitutes in addition to those which are particularlyset out in the accompanying drawings. Although the terms first, second,and the like, may be used herein to describe various elements, theseelements should not be limited by these terms. These terms are generallyonly used to distinguish one element from another.

Accordingly, preferred embodiments herein achieve a method forgenerating a partial virtual model of objects. The method includesdetecting, by a first electronic device, a video session with at leastone second electronic device. Further, the method includes receiving, bythe first electronic device, a first set of objects displayed in atleast one preview frame of the first electronic device and a second setof objects displayed in at least one second preview frame of the secondelectronic device. Further, the method includes determining, by thefirst electronic device, at least one first object from the first set ofobjects to be mapped to at least one second object from the second setof objects based on environmental observations in the video session.Further, the method includes predicting, by the first electronic device,a plurality of attributes of at least one visible portion of the atleast one first object and the at least one second object by mapping theat least one first object with the at least one second object. Further,the method includes providing, by the first electronic device, theplurality of predicted dimensions of the at least one visible portionthe at least one first object and the at least one second object asinput to a TOF sensor to generate the partial virtual model of the onefirst object and the at least one second object.

Unlike conventional methods and systems, the proposed method does notrequire multiple capture of objects from different camera positions, andcapture only visible portions of the objects during the communicationsession (e.g., video call, chat session, an online streaming session,online conferencing session, or the like). The visible portions arescanned to intelligently create a three dimensional (3D) model. Thusresulting in a low resource usage (e.g., CPU power cycles, battery,memory, and the like) of the electronic device for creating the 3Dmodel.

In the proposed method, there is no prerequisite required for any typeof data for creating the 3D model. The electronic device considers realtime dimensions of the object using the TOF sensor. Existing TOF sensorgenerates a complete 3D model even need of complete model is notrequired. There is no mechanism of generating only partial model ofobjects as required by the user. Hence, in the proposed method, onlyrequired portions of objects are mapped based on compatibility togenerate the partial model. In the proposed method, the TOF sensor isused to create a 3D depth map for only required body part of the firstobject and the second object to predict the compatibility of the firstobject and the second object. This resulting the low resource usage ofthe electronic device for creating the 3D model.

The proposed method does not require any special efforts required from auser to predict dimensional mapping of the first object and the secondobject while creating the partial 3D model. In case, if any dimensionwhich is predicted to be important could not be captured from framespresent in the video session, the method may be used to predict thatdimension using correction assumption technique.

In the proposed method, the electronic device utilizes only the requiredportion of the object to be used for 3D modelling and generate quick andpartial 3D model of the object required for processing. The electronicdevice only captures the visible portion of the object and can predictthe hidden dimensions with quite precision and accuracy.

Referring now to the drawings, and more particularly to FIGS. 2A through9B, there are shown preferred embodiments.

FIGS. 2A to 2F are example scenario in which a first electronic devicegenerates a partial virtual model of objects, according to variousembodiments of the disclosure.

The object may be, for example, but not limited to a product and anotheruser. The product may be, for example, but not limited to a chair, sofa,wheelchair, television, a refrigerator, a washing machine, and aninternal component of an electrical item or the like. The firstelectronic device (100 a) is in a video session with a second electronicdevice (100 b). The first electronic device (100 a) and the secondelectronic device (100 b) may be, for example, but not limited to asmart phone, a Personal Digital Assistant (PDA), a tablet computer, alaptop computer, an Internet of Things (IoT), a virtual reality device,an immersive system, and a smart watch.

Referring to FIG. 2A to 2F, the first electronic device (100 a) isconfigured to receive a first set of objects displayed in a previewframe of the first electronic device (100 a) and a second set of objectsdisplayed in a second preview frame of the second electronic device (100b). The first set of objects are in proximity to the first electronicdevice (100 a) and the first preview frame is displayed in a field ofview of a camera of the first electronic device (100 a). Similarity, thesecond set of objects are in proximity to the second electronic device(100 b) and the second preview frame is displayed in a field of view ofa camera of the second electronic device (100 b).

In another embodiment, the first electronic device (100 a) is configuredto receive the first set of objects displayed in the preview frame ofthe first electronic device (100 a) and a second set of objects isalready stored in the first electronic device (100 a) as an image or agroup of images.

Further, the first electronic device (100 a) may be configured todetermine a first object from the first set of objects to be mapped to asecond object from the second set of objects based on environmentalobservations in the video session. Further, the first electronic device(100 a) may be configured to predict a plurality of attributes ofvisible portion of the first object and the second object by mapping thefirst object with the second object. The plurality of attributes may be,for example, but not limited to a height of the object, a width of theobject, and a length of the object.

Further, the first electronic device (100 a) is configured to determinea structural component of the first object and a structural component ofthe second object. The structural component is any part of a frameworkof the first object or the second object. In other words, the structuralcomponent may be a base skeleton which is determined by includingdifferent required dimensions of the first object or the second object.It will have multiple dimensions for the first object or the secondobject, otherwise the structural component may be represented using onedimensional (1D) line model. Further, the first electronic device (100a) may be configured to map an intersection point of the structuralcomponent of the first object with the structural component of thesecond object by modifying a size of the first object and the secondobject. Further, the first electronic device (100 a) may be configuredto predict the plurality of attributes of visible portion of the firstobject and the second object based on the intersection point.

Further, the first electronic device (100 a) may be configured toacquire depth information associated with the first object and thesecond object. The depth information associated with the first objectmay be determined from a position of the first electronic device (100 a)and the depth information associated with the second object isdetermined from a position of the second electronic device (100 b).Consider, in an example, the D1 is a depth of first point on the firstobject from the smart phone and D2 is a depth of a last point of thefirst object from the smart phone then D2-D1 is the length of the firstobject in a plane. Thus, the length measured is independent of how farthe object is placed from the smart phone, and will remain same everytime it is measured through the sensor (e.g., depth sensor).

Further, the first electronic device (100 a) may be configured toprovide the plurality of predicted dimensions of the visible portion thefirst object and the second object and the acquired depth informationassociated with the first object and the second object as input to a TOFsensor to generate the partial virtual model of the first object and thesecond object.

The first electronic device (100 a) may be configured to receive theenvironmental observations tracked in the video session. Further, thefirst electronic device (100 a) is configured to generate a sub-partialvirtual model of the first object based on the plurality of predicteddimensions of the portion of the first object and the environmentalobservations tracked in the video session. Further, the first electronicdevice (100 a) is configured to generate a sub-partial virtual model ofthe second object based on the plurality of predicted dimensions of theportion of the second object and the environmental observations trackedin the video session. Further, the first electronic device (100 a) maybe configured to generate the partial virtual model by mapping thesub-partial virtual model of the first object with the sub-partialvirtual model of the second object using the TOF sensor.

In another embodiment, the sub-partial virtual model of the first objectmay be generated by determining dimension of not visible part of thefirst object and applying a machine learning model on the plurality ofpredicted dimensions of the visible portion of the first object and thedimension of the not visible portion of the first object to generate thesub-partial virtual model of the first object.

In another embodiment, the dimension of not visible part of the firstobject may be determined by virtually creating axis of similarity forportions of the first object, fetching a dimension of not visible partof the first object by virtually creating axis of similarity forportions of the first object, and determining the dimension of notvisible part of the first object based on the fetched dimension of thenot visible part of the first object.

The sub-partial virtual model of the second object may be generated bydetermining dimension of not visible part of the second object, andapplying the machine learning model on the plurality of predicteddimensions of the visible portion of the second object and the dimensionof the not visible portion of the second object to generate thesub-partial virtual model of the second object.

The dimension of not visible part of the second object may be determinedby virtually creating axis of similarity for portions of the secondobject, fetching a dimension of not visible part of the second object byvirtually creating axis of similarity for portions of the second object,and determining the dimension of not visible part of the second objectbased on the fetched dimension of not visible part of the second object.

Referring to FIG. 2A, the user (202) of the first electronic device (100a) is calling to a store owner (204) of the second electronic device(100 b) to buy a wheelchair (206-210). After receiving the call, thestore owner (204) of the second electronic device (100 b) is asking somedetails to the user (202) of the first electronic device (100 a). Thedetails may be, for example, but not limited to what is your condition?What size are you? What type do you want? or health condition.

Referring to FIG. 2A, an environment observer service is running in thefirst electronic device (100 a) and the second electronic device (100b). The environment observer service triggers a TOF interactioncontroller (explained in FIG. 5B) of the first electronic device (100 a)to receive various types of the wheelchair (206-210) displayed in thepreview frame (212) of the first electronic device (100 a) and theposition of the user (202) displayed in the second preview frame (214)of the second electronic device (100 b).

In another example, the environment observer service triggers the TOFinteraction controller of the first electronic device (100 a) to receivevarious types of the wheelchair displayed in the preview frame (212) ofthe first electronic device (100 a) and acquires the position of theuser from a memory stored as the image.

Referring to FIG. 2B, the TOF interaction controller may be configuredto start interaction between various TOF sensor involved in the firstelectronic device (100 a) and the second electronic device (100 b).Further, the TOF interaction controller may be configured tointelligently sense which wheelchair (208) and the user (202) arerequired to be mapped across the TOF sensors.

Referring to FIG. 2C, an object sub-unit mapping controller (explainedin FIG. 5B) in the first electronic device (100 a) intelligently handlesvarious object mapping across frames of the video call to map thevarious parts (216 e, 216 f, 216 g, and 216 h) of the wheelchair and thevarious parts (216 a, 216 b, 216 c and 216 d) of the user such as one toone, one to many or many to many in the frame. The various parts of thewheelchair may be, for example, but not limited to a user sitting part,a rest, an arm rest. The various parts of the users may be, for example,but not limited to legs, arms,

Referring to FIG. 2D, an object mapping controller (explained in FIG.5B) of the first electronic device (100 a) verifies the dimensionalsuitability of the wheelchair and the user predicted to be mapped witheach other along with which dimensions are required.

Referring to FIG. 2E, the first electronic device (100 a) intelligentlycreates the partial 3D model (218 and 220) of only those dimensionswhich are required to map. It also includes the predicted hiddenmeasurements which are not visible. The partial 3D models saves us CPUprocessing cycles, hardware usage, battery usage or the like

Referring to FIG. 2F, the first electronic device (100 a) virtuallyplaces the predicted objects onto each other in a suitable position andshares the intelligent display to the user (202) of the first electronicdevice (100 a).

FIG. 3 is an example scenario in which identifying points of thestructural intersection is depicted, according to an embodiment of thedisclosure.

Referring to FIG. 3 , the object mapping controller works on the basisof placing the object predicted to be mapped on each on the basis ofusage of the object and then strategically change various dimensions ofone of the objects to detect a hindrance of their structural unit. Thisdefining those dimensions as the one required to map the objects. Panel“a” of FIG. 3 depicts a structural mapping of the object (302) and panel“b” of FIG. 3 depicts a basic human sitting structure (304). Panel “c”of FIG. 3 depicts that gradually changing the size of each part of anyobject and the basic human sitting structure (306) in which structurallines of the basic human sitting structure (306) intersects withstructural lines of objects to identify points (308, 310, 312 and 314)of structural intersection as shown in panel “d” of FIG. 3 .

In other words, the first electronic device (100 a) may obtain a basicstructure of each object and place both object structure onto eachother. Further, the first electronic device (100 a) may keep any onestructure constant and increase/decrease the size of other structuregradually in all direction till the structures starts gettingintersected, such that intersected points are the dimensions requiredfor the mapping of both the objects.

In the proposed methods, by using the required 3D model of the secondobject, the first electronic device (100 a) may only capture thedimensions which is required for the second object mapping. Referring toFIG. 3 , the first electronic device (100 a) only captures the backportion of the user, a hip portion of the user, and a leg portion of theuser are used here. Further, the partial 3D model may be created usingonly the visible sides and using predicted hidden sides dimensions ofthe object. This resulting less usage of resources.

FIG. 4 is an example scenario in which determining attributes of notvisible part of an object is depicted for generating a partial virtualmodel of the object, according to an embodiment of the disclosure.

Referring to FIG. 4 , Panel “a” of FIG. 4 depicts the dimensions (402,404, 406, 408, 410, and 412) predicted to be required by the objectmapping controller and the object view in the video call is depictedwith the dimension (414) as shown in panel “b” of FIG. 4 .

Panel “c-e” of FIG. 4 , the first electronic device (100 a) virtuallycreates axis of similarity for portions of the first object (416) on thebasis of what percent of data shown in the frame are similar across bothsides of the axis. Further, the first electronic device (100 a) mayfetch a dimension of the not visible part of the second object byvirtually creating axis of similarity for portions of the second object(418). Further, the first electronic device (100 a) may determine thedimension of the not visible part of the second object based on thefetched dimension of the not visible part of the second object, suchthat the the first electronic device (100 a) may generate the partialvirtual model (420) of the object.

FIG. 5A shows various hardware components of a first electronic deviceor a second electronic device, according to an embodiment of thedisclosure.

Referring to FIG. 5A, the first electronic device (100 a) or the secondelectronic device (100 b) may include a processor (502), a communicator(504), a memory (506), a sensor (508), a partial virtual modelcontroller (510), a machine learning controller (512), and a camera(514). The sensor (508) may be, for example, but not limited to a TOFsensor and a depth sensor.

The processor (502) may be coupled with the communicator (504), thememory (506), the sensor (508), the partial virtual model controller(510), the machine learning controller (512), and the camera (514).

In an embodiment, the partial virtual model controller (510) may beconfigured to detect that the first electronic device (100 a) is in thevideo session with second electronic device (100 b).

Based on detecting the video session, the sensor (508) triggers theenvironment observer service running in the first electronic device (100a) and the second electronic device (100 b). The environment observerservice processes various data factors such conversational data, facialexpression or the like.

The sensor (508) may be configured to receive the first set of objectsdisplayed in preview frame of the first electronic device (100 a) andthe second set of objects displayed in the second preview frame of thesecond electronic device (100 b).

Further, the partial virtual model controller (510) may be configured tostart interaction between various sensors (508) involved in the firstelectronic device (100 a) and the second electronic device (100 b).Further, the partial virtual model controller (510) may be configured tointelligently sense which objects are required to be mapped across thesensor (508) of the first electronic device (100 a) and the secondelectronic device (100 b).

Further, the partial virtual model controller (510) in the firstelectronic device (100 a) intelligently handles various object mappingacross frames of the video session to map objects such as one to one,one to many or many to many in a video frame. The partial virtual modelcontroller (510) may verify the dimensional suitability of the objectspredicted to be mapped with each other along with which dimensions arerequired. Further, the partial virtual model controller (510) mayintelligently create the partial 3D model of only those dimensions whichare required to map. The partial 3D model may also include the predictedhidden measurements which are not visible. The partial virtual modelcontroller (510) may virtually place the predicted objects onto eachother in a suitable position and shares the intelligent display to theuser of the first/second electronic device (100 a or 100 b).

Further, the partial virtual model controller (510) may assume thedimension of those portions of the object that are not shown in anyframe of the video but are predicted to be important for mapping of 2objects. The machine learning controller (512) may train the machinelearning model includes variable parameter such as user's expressionacross the first and second electronic devices (100 a and 100 b),conversation snippets, etc., to predict the best result of theinteraction.

The processor (502) may be configured to execute instructions stored inthe memory (506) and to perform various processes. The communicator(504) may be configured for communicating internally between internalhardware components and with external devices via one or more networks.

The memory (506) may also store instructions to be executed by theprocessor (502). The memory (506) may include non-volatile storageelements. Examples of such non-volatile storage elements may includemagnetic hard discs, optical discs, floppy discs, flash memories, orforms of electrically programmable memories (EPROM) or electricallyerasable and programmable (EEPROM) memories. In addition, the memory(506) may, in some examples, be considered a non-transitory storagemedium. The term “non-transitory” may indicate that the storage mediumis not embodied in a carrier wave or a propagated signal. However, theterm “non-transitory” should not be interpreted that the memory (506) isnon-movable. In some examples, the memory (506) may be configured tostore larger amounts of information than the memory. In certainexamples, a non-transitory storage medium may store data that may, overtime, change (e.g., in Random Access Memory (RAM) or cache).

Further, at least one of a plurality of hardware components may beimplemented through an artificial intelligent (AI) model. A functionassociated with AI may be performed through the non-volatile memory, thevolatile memory, and the processor (502). The processor (502) mayinclude one or a plurality of processors. At this time, one or aplurality of processors may be a general purpose processor, such as acentral processing unit (CPU), an application processor (AP), or thelike, a graphics-only processing unit such as a graphics processing unit(GPU), a visual processing unit (VPU), and/or an AI-dedicated processorsuch as a neural processing unit (NPU).

The one or a plurality of processors may control the processing of theinput data in accordance with a predefined operating rule or artificialintelligence (AI) model stored in the non-volatile memory and thevolatile memory. The predefined operating rule or artificialintelligence model is provided through training or learning.

Here, being provided through learning means that, by applying a learningalgorithm to a plurality of learning data, a predefined operating ruleor AI model of a desired characteristic is made. The learning may beperformed in a device itself in which AI according to an embodiment isperformed, and/o may be implemented through a separate server/system.

The AI model may consist of a plurality of neural network layers. Eachlayer has a plurality of weight values, and performs a layer operationthrough calculation of a previous layer and an operation of a pluralityof weights. Examples of neural networks include, but are not limited to,convolutional neural network (CNN), deep neural network (DNN), recurrentneural network (RNN), restricted Boltzmann Machine (RBM), deep beliefnetwork (DBN), bidirectional recurrent deep neural network (BRDNN),generative adversarial networks (GAN), and deep Q-networks.

The learning algorithm is a method for training a predetermined targetdevice (e.g., a robot) using a plurality of learning data to cause,allow, or control the target device to make a determination orprediction. Examples of learning algorithms include, but are not limitedto, supervised learning, unsupervised learning, semi-supervisedlearning, or reinforcement learning.

Although FIG. 5A shows various hardware components of the firstelectronic device (100 a) or the second electronic device (100 b) but itis to be understood that other embodiments are not limited thereon. Inother embodiments, the first electronic device (100 a) or the secondelectronic device (100 b) may include less or more number of components.Further, the labels or names of the components are used only forillustrative purpose and does not limit the scope of the disclosure. Oneor more components may be combined together to perform same orsubstantially similar function to generate the partial virtual model ofobjects.

FIG. 5B shows various hardware components of a partial virtual modelcontroller, according to an embodiment of the disclosure.

Referring to FIG. 5B, the partial virtual model controller (510) mayinclude a TOF interaction controller (514 a), an object mappingcontroller (514 b), an object sub-unit mapping controller (514 c), and anon-visible dimension correction controller (514 d).

The TOF interaction controller (514 a) may be configured to receive thefirst set of objects displayed in the preview frame of the firstelectronic device (100 a) and the second set of objects displayed in thesecond preview frame of the second electronic device (100 b). Further,the TOF interaction controller (514 a) may be configured to startinteraction between various sensors (508) involved in the firstelectronic device (100 a) and the second electronic device (100 b).Further, the TOF interaction controller (514 a) may be configured tointelligently sense which objects are required to be mapped across thesensors (508) of the first electronic device (100 a) and the secondelectronic device (100 b).

Further, the object sub-unit mapping controller (514 c) mayintelligently handle various object mapping across frames of the videosession to map objects in the video frame. The object mapping controller(514 b) may verify the dimensional suitability of the objects predictedto be mapped with each other along with which dimensions are required.Further, the partial virtual model controller (510) may intelligentlycreate the partial 3D model of only those dimensions which are requiredto map. The partial 3D model may also include the predicted hiddenmeasurements which are not visible. The partial virtual model controller(510) may virtually place the predicted objects onto each other in asuitable position and shares the intelligent display to the user of thefirst electronic device (100 a).

Further, the non-visible dimension correction controller (514 d) mayassume the dimension of those portions of the object that are not shownin any frame of the video but are predicted to be important for mappingof two objects.

FIGS. 6A and 6B are example sequence diagram illustrating step by stepprocess for generating the partial virtual model of objects, accordingto various embodiments of the disclosure.

Referring to FIGS. 6A and 6B, at S602 a and S602 b, the TOF interactioncontroller (514 a) may start interaction between various sensors (508)involved in the first smart phone and the second smart phone. At S604 aand S604 b, the TOF interaction controller (514 a) may send a requestfor obtaining the frame from the camera (514) of the first smart phoneand the second smart phone, respectively. At, S606 a and S606 b, the TOFinteraction controller (514 a) may receive the first set of objectsdisplayed in the preview frame of the first smart phone and the secondset of objects displayed in the second preview frame of the second smartphone, respectively.

At S608, the object mapping controller (514 b) may verify thedimensional suitability of the objects predicted to be mapped with eachother along with which dimensions are required. At S610, the objectsub-unit mapping controller (514 c) intelligently handles various objectmapping across frames of the video session to map objects in the videoframe. At S612 and 5614, the TOF interaction controller (514 a) may beconfigured to intelligently sense which objects are required to bemapped across the sensors (508) of the first electronic device (100 a)and the second electronic device (100 b).

At S616, the TOF interaction controller (514 a) of the second smartphone request the required dimension of the first object. At S618, theTOF interaction controller (514 a) of the first smart phone shared therequired dimension of the first object to the TOF interaction controller(514 a) of the second smart phone based on the request. At S620, the TOFinteraction controller (514 a) of the first smart phone is configuredsend the dimension of the first object and the dimension of the secondobject to the object sub-unit mapping controller (514 c). At S622, theobject sub-unit mapping controller (514 c) is configured tointelligently sense which objects are required to be mapped across thesensors (508) of the first electronic device (100 a) and the secondelectronic device (100 b).

At S624, the non-visible dimension correction controller (514 d) mayassume the dimension of those portions of the object that are not shownin any frame of the video but are predicted to be important for mappingof two objects.

At S626, the partial virtual model controller (510) may intelligentlycreate the partial 3D model of only those dimensions which are requiredto map. The partial 3D model may also include the predicted hiddenmeasurements which are not visible. At S628, the partial virtual modelcontroller (510) may virtually place the predicted objects onto eachother in a suitable position and shares the intelligent display to theuser of the first electronic device (100 a).

FIGS. 7A and 7B are an example flow chart illustrating a method forgenerating a partial virtual model of objects, according to anembodiment of the disclosure.

Referring to FIGS. 7A and 7B, in a method S700, the operations S702-S718are performed by the partial virtual model controller (510). At S702,the method may include detecting that the first electronic device (100a) is in the video session with the second electronic device (100 b). AtS704, the method may include receiving the first set of objectsdisplayed in preview frame of the first electronic device (100 a) andthe second set of objects displayed in second preview frame of thesecond electronic device (100 b). At S706, the method may includedetermining the first object from the first set of objects to be mappedto the second object from the second set of objects based on theenvironmental observations in the video session.

At S708, the method may include predicting the plurality of attributesof the visible portion of the first object and the second object bymapping the first object with the second object. At S710, the method mayinclude acquiring the depth information associated with the first objectand the second object. At S712, the method may include providing theplurality of predicted dimensions of the visible portion the firstobject and the second object as input to the TOF sensor (508) togenerate the partial virtual model of the first object and the secondobject.

At S714, the method may include receiving the environmental observationstracked in the video session. At S716, the method may include generatingthe sub-partial virtual model of the first object based on the pluralityof predicted attributes of the portion of the first object and theenvironmental observations tracked in the video session. At S718, themethod may include generating the sub-partial virtual model of thesecond object based on the plurality of predicted attributes of theportion of the second object and the environmental observations trackedin the video session. At S720, the method may include generating thepartial virtual model by mapping the sub-partial virtual model of thefirst object with the sub-partial virtual model of the second objectusing the sensor.

FIG. 7C is a flow diagram illustrating various operations for generatinga sub-partial virtual model of a first object based on a plurality ofpredicted attributes of a portion of the first object and anenvironmental observations tracked in a video session, according to anembodiment of the disclosure.

Referring to 7C, in operation S716, at 716 a, the method may includedetermining the attributes of the not visible part of the first object.At 716 b, the method may include applying the machine learning model onthe plurality of predicted attributes of the visible portion of thefirst object and the portion of the not visible portion of the firstobject to generate the sub-partial virtual model of the first object.

FIG. 7D is a flow diagram illustrating various operations for generatingthe sub-partial virtual model of the second object based on theplurality of predicted attributes of the portion of the second objectand the environmental observations tracked in the video session,according to an embodiment of the disclosure.

Referring to FIG. 7D, in operation S718, at 718 a, the method mayinclude determining dimension of not visible part of the second object.At 718 b, the method may include applying the machine learning model onthe plurality of predicted attributes of the visible portion of thesecond object and the predicted attributes of the not visible portion ofthe second object to generate the sub-partial virtual model of thesecond object.

FIG. 7E is a flow diagram illustrating various operations for predictingthe plurality of attributes of the visible portion of the first objectand the second object by mapping the first object with the secondobject, according to an embodiment of the disclosure.

Referring to FIG. 7E, in operation S708, at 708 a, the method mayinclude determining the structural component of the first object and thestructural component of the second component. At 708 b, the method mayinclude mapping the intersection point of the structural component ofthe first object with the structural component of the second object bymodifying a size of the first object and the second object. At 708 c,the method may include predicting the plurality of attributes of visibleportion of the first object and the second object based on theintersection point.

FIG. 7F is a flow diagram illustrating various operations fordetermining the dimension of not visible part of the first object,according to an embodiment of the disclosure.

Referring to FIG. 7F, in operation S716 a, at 716 aa, the method mayinclude virtually creating the axis of similarity for portions of thefirst object. At 716 ab, the method may include fetching the dimensionof not visible part of the first object by virtually creating axis ofsimilarity for portions of the first object. At 716 ac, the method mayinclude determining the dimension of not visible part of the firstobject based on the fetched dimension of not visible part of the firstobject.

FIG. 7G is a flow diagram illustrating various operations fordetermining the dimension of not visible part of the second object,according to an embodiment of the disclosure.

Referring to FIG. 7G, in operation S718 a, at 718 aa, the method mayinclude virtually creating the axis of similarity for portions of thesecond object. At 718 ab, the method may include fetching the dimensionof the not visible part of the second object by virtually creating axisof similarity for portions of the second object. At 718 ac, the methodmay include determining the dimension of the not visible part of thesecond object based on the fetched dimension of not visible part of thefirst object.

The various actions, acts, blocks, steps, or the like in the S700 (S708,S716, S716 a, S718, and S718 a) may be performed in the order presented,in a different order or simultaneously. Further, in some embodiments,some of the actions, acts, blocks, steps, or the like may be omitted,added, modified, skipped, or the like without departing from the scopeof the disclosure.

FIG. 8A to FIG. 8E are example scenario in which a virtual assistedshopping using the TOF sensors is depicted, according to variousembodiments of the disclosure.

Referring to FIG. 8A, the user (804) of the first electronic device (100a) initiates the video call to a furniture shop and the user of thefirst electronic device (100 a) requests to buy the sofa (806). The userof the first electronic device (100 a) shows him/her a space (808) inthe empty room where the user intends to put the sofa. Further, the shopowner shows the sofa (806) to the user (804) of the first electronicdevice (100 a) and asks where does the user wants to place it. Referringto FIGS. 8B to 8D, on the basis of conversation over the video call, thefirst electronic device (100 a) selects the objects (i.e., user and sofaand sofa and space in the living room) to be mapped and the requirementof dimensional intelligence is triggered automatically by the proposedmethod. The objects are selected automatically to be mappeddimensionally (808, 810, 812, and 814). In this case, multiple mappingis observed by the proposed method, 1) Sofa to Placement Location and 2)Sofa to User.

Intelligently few automatically deduced required dimensions (e.g.,length, width, height) of the sofa is mapped with the requireddimensions (e.g., length, width, height) of the location at which theuser (802) wishes to place the sofa (806). Also for second mapping fewother automatically deduced required dimensions of the sofa (806) ismapped with the required dimensions (e.g., height, width, or the like)of the user. Further, the partial virtual model controller (510) maytake inputs from the TOF sensor in form of dimension data and themachine learning model output. It will map the best fit furniture (816)in the living room.

FIGS. 9A and 9B are example scenario in which an assisted fittingmeasurement using the TOF sensors is depicted, according to variousembodiments of the disclosure.

Referring to FIG. 9A, at home where a hardware (H/W) failure isoccurred. The video call is placed from a user of the first electronicdevice (100 a) to an inventory (904). Using the TOF interactioncontroller (514 a), all parameters are measured. The parameters may be,for example, but not limited to an inner diameter (ID), an outerdiameter (OD), a wall thickness, thread size. The parameters is sharedwith the partial virtual model controller (510), wherein the parametersis shared along with the call data. The machine learning controller(512) will take inputs from the mechanical system in place and otherenvironmental factors like wear and tear, rust etc.

Referring to FIG. 9B, at the other end of the video call, the TOFinteraction controller (514 a) will be triggered by backgroundenvironment observer service. Using the TOF interaction controller (514a), all parameters of the inventory (904) are measured. The parametersare checked and mapped to earlier stored parameters and shared with thepartial virtual model controller (510). The inputs from the past salesabout the part, new models other environmental factors will be fed tothe machine learning controller (512).

Further, the partial virtual model controller (510) will take inputsfrom the TOF interaction controller (514 a) in form of dimension dataand the machine learning model output. It will predict the best fit part(906) to be replaced in the existing system as shown in FIGS. 9A and 9B.

The embodiments disclosed herein may be implemented using at least onesoftware program running on at least one hardware device and performingnetwork management functions to control the elements.

While the disclosure has been shown and described with reference tovarious embodiments thereof, it will be understood by those skilled inthe art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the disclosure as definedby the appended claims and their equivalents.

What is claimed is:
 1. A method for generating a virtual model ofobjects, comprising: detecting, by a first electronic device, acommunication session with a second electronic device; obtaining a firstset of objects displayed on the first electronic device and a second setof objects displayed on the second electronic device based on thedetection of the communication session; determining a first object fromthe first set of objects to be mapped to a second object from the secondset of objects; predicting attributes of visible portions of the firstobject and the second object by mapping the first object to the secondobject; obtaining depth information related to the first object and thesecond object; and generating a virtual model of the first object andthe second object based on the predicted attributes of the visibleportions of the first object and the second object and the depthinformation related to the first object and the second object.
 2. Themethod of claim 1, wherein the first set of objects is displayed in afirst preview frame of the first electronic device, and wherein thesecond set of objects is displayed in a second preview frame of thesecond electronic device.
 3. The method as claimed in claim 1, whereinthe first set of objects is displayed in a first preview frame of thefirst electronic device, and wherein the second set of objects ispre-stored in a memory of the first electronic device.
 4. The method ofclaim 1, wherein the first object to be mapped to the second object isdetermined based on environmental observation information tracked in thecommunication session.
 5. The method of claim 4, wherein the generatingof the virtual model comprises: generating a first sub-partial virtualmodel of the first object based on the predicted attributes of the firstobject and the tracked environmental observation information; generatinga second sub-partial virtual model of the second object based on thepredicted attributes of the second object and the tracked environmentalobservation information; and mapping the first sub-partial virtual modelof the first object with the second sub-partial virtual model of thesecond object to generate the virtual model.
 6. The method of claim 5,wherein the generating of the first sub-partial virtual model of thefirst object comprises: determining an attribute of an invisible portionof the first object; and applying a machine learning model to thepredicted attributes of the visible portions of the first object and thedetermined attribute of the invisible portion of the first object togenerate the first sub-partial virtual model of the first object.
 7. Themethod of claim 6, wherein the determining of the attribute of theinvisible portion of the first object comprises: generating virtually anaxis of similarity for portions of the first object; fetching adimension of the invisible portion of the first object based on thesimilarity axis; and determining a dimension of the invisible portion ofthe first object based on the fetched dimension of the invisible portionof the first object.
 8. The method of claim 5, wherein the generating ofthe second sub-partial virtual model of the second object comprises:determining an attribute of an invisible portion of the second object;and applying a machine learning model to the predicted attributes of thevisible portion of the second object and the determined attribute of theinvisible portion of the second object to generate the secondsub-partial virtual model of the second object.
 9. The method of claim8, wherein the determining of the attribute of the invisible portion ofthe second object comprises: generating virtually an axis of similarityfor portions of the first object; fetching a dimension of the invisibleportion of the second object based on the similarity axis; anddetermining a dimension of the invisible portion of the second objectbased on the fetched dimension of the invisible portion of the secondobject.
 10. The method of claim 1, wherein the predicting of theattributes of visible portions includes: determining a first structuralcomponent of the first object and a second structural component of thesecond object; mapping at least one intersection point of the firststructural component of the first object with the second structuralcomponent of the second object by modifying a size of at least one ofthe first object and the second object; and predicting the attributes ofthe visible portions of the first object and the second object based onthe at least one intersection point.
 11. An electronic devicecomprising: a communicator configured to communicate with an externaldevice; a camera configured to capture a first set of objects comprisinga first object; a sensor configured to acquire a first depth informationrelated to the first object; and at least one processor electricallyconnected to the communicator, the camera, and the sensor, wherein theat least one processor is configured to: detect a communication sessionwith the external device, obtain the captured first set of objects and asecond set of objects displayed on the external device based on thedetection of the communication session, determine the first object fromthe first set of objects to be mapped to a second object from the secondset of objects, predict attributes of visible portions of the firstobject and the second object by mapping the first object to the secondobject, obtain a first depth information related to the first objectfrom the sensor, obtain a second depth information related to the secondobject, and generate a virtual model of the first object and the secondobject based on the predicted attributes of the visible portions of thefirst object and the second object, the first depth information and thesecond depth information.
 12. The electronic device of claim 11, whereinthe at least one processor is further configured to obtain the first setof objects is displayed in a first preview frame of the electronicdevice, and wherein the second set of objects is displayed in a secondpreview frame of the external device.
 13. The electronic device of claim11, further comprising a memory, wherein the at least one processor isfurther configured to obtain the first set of objects is displayed in afirst preview frame of the electronic device, and wherein the second setof objects is pre-stored in the memory.
 14. The electronic device ofclaim 11, wherein the at least one processor is further configured todetermine the first object to be mapped to the second object based onenvironmental observation information tracked in the communicationsession.
 15. The electronic device of claim 14, wherein the at least oneprocessor is further configured to: generate a first sub-partial virtualmodel of the first object based on the predicted attributes of the firstobject and the tracked environmental observation information, generate asecond sub-partial virtual model of the second object based on thepredicted attributes of the second object and the tracked environmentalobservation information, and map the first sub-partial virtual model ofthe first object with the second sub-partial virtual model of the secondobject to generate the virtual model.